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VICTORY  Architecture  Position  Service  Validation 

Experiments 


1.0  Introduction 

This  document  describes  a  set  of  validation  experiments  and  will  provide 
verification  and  validation  of  the  VICTORY  1.0  Architecture  Position  (GPS)  service 
specifications.  The  Position  service  when  implemented  correctly  and  with  the 
standard  format  specified  by  the  VICTORY  1.0  will  provide  position  data  to  the  VDB 
(Victory  Data  Bus)  as  shown  in  Figure  l.The  main  components  and  interfaces  being 
evaluated  on  the  VDB  (VICTORY  Data  Bus)  include 
o  Position  data  VDM  (VICTORY  Data  Message) 
o  VDB  Position  Data  Interface 


Figure  1:  VICTORY  Services  network  as  implemented  in  the  VIDS 

The  experiments  will  evaluate  these  interface  specifications  by  integrating  software 
clients  and  services  developed  using  the  specifications,  and  evaluating  the  resulting 
functional  behavior  and  performance.  The  TARDEC  Vehicle  Electronics  and 
Architecture  (VEA)  group  executed  this  set  of  additional  validation  experiments, 
utilizing  their  VICTORY  Interoperability  &  Development  System  Integration 
Laboratory  (VIDS) 
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2.0  Experiment  Goals 

o  Evaluate  the  completeness  and  unambiguousness  of  the  individual 
interface  specifications  of  each  included  service. 

o  Investigate  the  resulting  functional  performance  of  reference 
implementations  of  the  interfaces  integrated  with  representative 
hardware. 

3.0  Experiment  Design 

The  experiments  in  this  set  all  leverage  a  common  hardware  and  software  design. 
The  Physical/Logical  block  diagram  for  testing  the  Position  service  is  shown  in 
Figure  2.  The  two  test  tools  used  for  managing  and  monitoring  the  position  VDM’s 
are, 

a.  WiresharkVDB  plug-in 

b.  Position  Service  Management  Interface 


3.1  Wireshark  VDB  Plug-in 

A  custom  dissector  plug-in  for  Wireshark  version  1.2.8  is  developed  for  the  VIDS  lab 
and  is  used  as  a  tool  for  testing  and  monitoring  VDM’s.  This  dissector  captures  UDP 
VICTORY  Data  Messages  (VDMs)  and  breaks  them  down  into  their  specific  header 
and  data  fields  as  show  in  Figure  3.  It  also  provides  a  filter  to  look  for  VDM 
messages  and  the  ability  to  log  captured  VDMs  to  a  formatted  text  file.  Each  properly 
formatted  VDM  packet  contains  a  "Header”  field  and  a  "Data”  field,  which  can  be 
expanded  upon  as  described  in  the  following  sections. 
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Figure_3:  Wireshark  VDB  Plug-in  Screen  Shot 


3.2  Terminal  Client  and  GUI  Clients  for  Position  VDM  Management 


In  addition  to  the  Wireshark  plug-in  to  view  the  data  and  header,  two  clients  are 
developed  to  manage  the  VDM’s.  One  is  a  command  line  client  and  the  other  is  a 
GUI  based  client  as  shown  in  Figure  4  &  Figure  5.  Both  of  these  clients  perform 
the  same  VDM  control  functionality,  i.e.  to  enable/disable,  set  data  rates,  set 
update  period,  etc. 


Figure  4:  Terminal  Client  for  Position  VDM  Management 
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Figure  5:  GUI  Client  for  Position  VDM  Management 


4.0  Specifications  Evaluated: 

The  VICTORY  Architecture  1.0  standards  are  specified  in  the  Version  1.0  document 
released  on  July  29,  2011.  The  following  are  the  standards  that  were  evaluated 
o  VICTORY  Data  Management  Interface  <10008-20110729,  Pro> 
o  SOAP  Compliance  and  Standards  <10001-20110729,  Pro> 
o  Application  Layer  Data  Encoding  <20001-20110729,  Pro> 
o  Application  Layer  Message  Encapsulation  <20002-20110729,  Pro> 
o  Timestamp  Format  for  Application  Layer  Data  <20004-20110729,  Pro  > 
o  Position  Data  Interface  <20009-20110729,  Pro> 
o  Position  Management  <15003-20110729,  Pro> 
o  Position  Interface  Complex  Types  <20014-20110729,  Pro> 
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5.0  List  of  Experiments 

This  experiment  set  is  composed  of  the  following  procedures: 

o  Position  Management  Static  Parameters 
o  Position  Data  Enabled 
o  Position  Data  Validity 
o  Update  Period 
o  Sequence  number  continuity 
o  System  Resource  Usage 
o  End-to-End  Latency 


6.0  Procedure  1:  Position  Management  Static  Parameters 

The  position  management  interface  must  maintain  a  set  of  static  parameters  and 
supply  them  to  the  client’s  request.  This  procedure  will  evaluate  whether  the 
position  management  service  can  reply  to  the  client’s  Get()  requests  on  all  static 
parameters.  The  list  of  static  parameters  includes:  position  data  management 
specific  parameters  (Geodetic  datum  and  nominal  data  available),  version 
information  management  specific  parameters  (interface  type,  interface  standard 
version  and  configuration  version)  and  VICTORY  data  management  specific 
parameters  (time  uncertainty  and  minimum  update  period). 

A  terminal  position  management  client  described  in  the  previous  section  3.0  is  used 
to  execute  Get()  commands  on  all  of  the  static  parameters  mentioned  above.  The 
results  of  those  Get()  commands  are  recorded  and  compared  to  the  specification  to 
determine  their  validity. 

Procedure  Details 

a.  Execute  a  get()  command  on  every  static  parameter 

b.  Write  down  the  received  values 

c.  Although  the  specification  doesn’t  declare  the  format  and  range  of  the  static 
parameters,  all  static  parameters  are  implicitly  assumed  to  have  the  same 
values  as  the  ones  in  the  ‘Static  Configuration  Settings’  file.  Therefore  the 
received  values  from  the  get()  request  should  be  compared  with  the  ‘Static 
Configuration  Settings’  file. 

Results 

•  The  received  values  from  the  get()  request  compared  with  the  'Static 

Configuration  Settings’  file  are  shown  in  Table  1  below,  the  values  from  the 
get()  compare  identically  with  the  settings  from  the  'Static  Configuration 
Settings’  file. 
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Static  parameters 

Values  received  by  client 

Values  stored  in  the  'Static 
Configuration  Settings’  file 

Interface  ID 

N/A 

N/A 

Source  ID  list 

N/A 

N/A 

Interface  type 

Position 

Position 

Interface  standard 
version 

1.0 

1.0 

Configuration  Version 

TARDEC_VEA_VIDS_1.0 

TARDEC_VEA_VIDS_1.0 

Geodetic  datum 

N/A 

N/A 

Timestamp 

uncertainty 

0 

0 

Minimum  update 
period 

0 

0 

Nominal  data  available 

0,  0,0 

0,  0,0 

Table  1:  Position  service  Static  Parameters  results 


7.0  Procedure  2:  Position  Data  Enabled 

Data  enabled  is  a  parameter  that  identifies  whether  the  position  data  interface  is 
sending  out  data.  This  procedure  validates  whether  the  Position  Management 
Interface  can  reply  to  GetQ  and  Set()  requests  on  data  enabled  and  whether  the  flow 
of  position  data  is  properly  controlled  by  the  data  enabled  parameter. 

This  procedure  starts  by  using  a  position  management  client  to  send  a  data  enabled 
SetQ  command  with  value  of  True  to  the  position  management  interface.  After 
processing  the  SetQ  command,  the  position  management  interface  is  supposed  to 
enable  the  flow  of  position  VDMs,  which  will  be  verified  using  the  Wireshark  VDM 
Plug-in  described  in  Section  3.0  with  appropriate  network  filters.  Then  the  position 
management  client  will  send  a  GetO  command  to  the  position  management 
interface  for  data  enabled  parameter,  with  an  expected  value  of  True  as  it  was  set 
previously.  Next  the  position  management  client  will  send  another  SetO  command 
on  data  enabled  with  the  value  of  False  this  time.  Wireshark  will  again  verify  that 
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there  is  no  longer  any  position  VDMs  sent  out.  The  position  management  client  will 
send  a  data  enabled  Get()  command  to  ensure  that  the  received  value  is  False.  After 
that  the  position  management  client  will  set  the  value  of  data  enabled  to  True  again, 
and  Wireshark  will  verify  that  position  VDMs  is  sent  out  also.  Finally,  the  position 
management  client  will  send  out  a  Get()  request  to  ensure  that  the  value  of  data 
enabled  is  True. 

Procedure  Details 

a.  Use  the  client  to  set  the  update  period  to  1  second 

b.  Execute  a  set()  command  on  data  enabled  property  to  True 

c.  Verify  on  Wireshark  that  position  data  is  being  sent  out  periodically  for  at 
least  10  update  periods 

d.  Execute  a  get()  command  on  data  enabled  property 

e.  Verify  that  the  received  value  is  1 

f.  Execute  a  set()  command  on  data  enabled  property  to  False 

g.  Verify  on  Wireshark  that  position  data  stops  sending  out  for  a  time  out  of  at 
least  10  update  periods 

h.  Execute  a  get()  command  on  data  enabled  property 

i.  Verify  that  the  received  value  is  0 

j.  Set  the  data  enabled  property  back  to  True 

k.  Verify  on  Wireshark  that  the  position  data  is  sent  out  periodically  for  at  least 
10  update  periods 

l.  Use  the  client  to  request  data  enabled  property  and  verify  that  it  is  1 

Results 

•  Both  Get[)  and  Set()  commands  work  on  the  data  enabled  setting. 

•  When  the  data  enabled  property  is  1  (True),  position  data  is  sent  out 
periodically. 

•  When  the  data  enabled  property  is  0  (False),  position  data  stops  sending  out. 

•  Overall,  the  outcome  matches  the  expected  response. 

8.0  Procedure  3:  Position  Data  Validity 

Before  sending  out  position  VDMs,  the  position  data  interface  must  populate  all 
necessary  fields  of  the  binary  header,  formulate  the  XML  payload  according  to  the 
position  data  schema  and  preserve  the  integrity  of  the  raw  data  sampled  from  any 
GPS  source.  This  procedure  will  determine  the  integrity  of  position  data  values  and 
the  validity  of  binary  header  and  XML  payload.  It  will  also  determine  if  position  data 
can  be  extracted  from  the  position  VDMs  in  an  unambiguous  way. 

In  this  procedure  simulated  position  data  will  be  generated  and  fed  to  the  position 
data  interface.  And  the  output  position  VDMs  will  be  analyzed  and  validated. 

Procedure  Details 

a.  Use  DAGR  data  input  to  feed  the  current  location 

b.  Run  Position  Server  to  read  DAGR  data  and  generate  Position  VDM 

c.  Start  a  client  connection  to  the  Position  Service  Management 
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d.  Execute  a  setQ  command  on  data  enabled  property  to  True 

e.  Run  the  Position  Data  Sink  for  an  hour  to  interpret  VDM  multicast  messages 
and  validate  the  data 

f.  The  Position  Data  Sink  will  write  the  validation  result  into  a  log  file 

Results 

•  All  messages  have  valid  binary  header. 

•  For  the  implementation  evaluated: 

-  The  position  values  received  and  extracted  from  the  VDB  matched  the 
values  that  the  service  encapsulated  and  sent  in  the  VDM. 

-  The  position  values  sent  out  by  the  data  service  is  identical  to  the 
DAGR  values. 


9.0  Procedure  4:  Update  Period 

The  position  management  service  must  allow  dynamic  GetQ  and  SetQ  functions  on 
update  period  parameter.  And  the  position  data  service  should  arrange 
transmission  delay  between  each  position  VDM  in  consistence  with  the  update 
period  parameter. 

The  position  management  client  will  set  the  update  period  to  different  values.  At 
each  value,  position  VDMs  will  be  collected  for  an  hour  and  analyzed  to  determine  if 
the  update  period  value  is  consistent  with  the  transmission  period  between  each 
position  VDM.  Consistency  is  a  subjective  criterion  in  this  case. 

Procedure  Details 

a.  Use  the  client  to  query  the  Position  Management  Service  for  minimum 
update  period 

b.  Do  a  setQ  command  to  change  the  update  period  to  its  minimum 

c.  GetQ  the  value  and  verify  it  was  set  correctly 

d.  Use  the  Position  Data  Sink  to  listen  to  position  multicast  stream  and  collect 
position  data  for  an  hour.  The  Position  Data  Sink  will  record  the  timestamps 
and  differences  between  successive  timestamps  into  a  file. 

e.  At  the  same  time,  use  Wireshark  with  the  appropriate  capture  filter  to 
capture  position  data  messages 

f.  Calculate  the  average  of  the  list  of  period  between  successive  timestamps 
from  the  record  file 

g.  Use  Wireshark  to  generate  average  packets/sec  statistic.  Its  inverse  will  be 
the  average  update  period. 

h.  Record  the  value  from  Wireshark 
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Results 

Wireshark  results  were  captured  for  a  period  of  lhour.  The  average  update  period 
for  the  VDM  message  set  for  a  lsec  update  period  is  calculated  and  show  below, 


Update  Period  Setting 

Average  value  update  for  VDM  message 
from  Wireshark 

1  sec 

1.000837923  sec 

Table  2:  Position  Service  Update  Period  results 


10.0  Procedure  5:  Sequence  number  continuity 

The  position  data  interface  must  tag  each  message  with  a  monotonically  increasing 
sequence  number  before  sending  it  out.  To  validate  the  sequence  number  of 
position  VDMs,  the  number  of  discontinuities  will  be  recorded.  Position  VDMs  will 
be  continuously  sent  out  for  an  hour,  and  a  position  data  sink  will  analyze  the 
continuity  of  sequence  numbers  of  all  position  VDMs. 

Procedure  Details 

a.  Use  the  client  to  query  for  the  minimum  update  period 

b.  Set  the  update  period  to  twice  the  minimum  period 

c.  Perform  a  getQ  command  to  verify  that  the  update  period  was  set  correctly 

d.  Use  the  Position  Data  Sink  to  check  if  the  sequence  number  increases 
monotonically 

e.  Allow  Position  Data  Sink  run  for  one  hour  and  record  sequence  number 
discontinuities 


Result 

Received  sequence  numbers  increases  monotonically  without  any  discontinuities. 


11.0  Procedure  6:  System  Resource  Usage 

This  procedure  will  investigate  the  computing  resources  required  to  interpret 
position  messages.  The  recommended  metrics  for  system  resource  usage  is  XML 
processing  time. 

Procedure  Details 

a.  Start  the  Position  Data  Service  at  a  reasonable  update  rate 

b.  Use  the  Position  Data  Sink  to  collect  values  of  processing  time  and  memory 
used  during  the  interpretation  and  validation  of  VDM  messages 

c.  Display  the  results  in  the  log  file  graphically 

Result 
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The  processing  time  for  creating  position  VDM  ranges  from  52,000  nsec,  to  91,000 
nsec,  with  some  sporadic  peaks  up  to  114,000  nsec  as  shown  in  Figure  6.  The 
average  XML  build  time  is  83,000  nsec  and  Std  Dev  of  43,840  nsec. 
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Figure  6:  VDM  build  time  for  Position  Service 

12.0  Procedure  7:  End-to-End  Latency 

The  accuracy  of  position  data  heavily  depends  on  the  latency  from  the  GPS  sensor 
output  to  the  received  corresponding  update  from  the  position  data  service.  This 
procedure  will  evaluate  the  end-to-end  latency  by  measuring  the  time  it  takes  to 
create  and  transfer  a  position  VDM  with  raw  data  received  from  a  position  sensor 
and  sunk  into  by  a  position  data  client.  The  raw  position  device  is  a  RMC 
(recommended  minimum  data  for  GPS)  string  output  from  a  serial  port  of  the  DAGR 
GPS  device. 

Procedure  Details 

a.  Configure  the  DAGR  to  broadcast  the  GPS  data  in  RMC  sentence.  The  Position 
Server  mentioned  in  Figure  1.0  receives  the  RMC  sentences  from  the  DAGR 
over  the  RS-232  connection. 

b.  Start  the  Position  Data  Service  at  the  minimum  update  period  (0ms  for  this 
reference  implementation),  the  data  service  will  parse  and  create  the 
Position  VDM’s  and  will  start  broadcasting  them. 

c.  Start  Wireshark  Position  Data  Sink.  It  will  continually  receive  position  VDM 
until  it  finds  the  same  GPS  values  inside  the  XML  payload.  While  doing  that,  it 


♦ 


Average  =  83,000nsec 
Standard  Deveiation  =  43,000nsec 
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keeps  track  of  2  timestamps:  one  when  the  Position  Data  Service  just  finishes 
parsing  and  creating  the  VDM’s  and  another  when  the  updated  VDM  is 
received.  The  end  to  end  latency  of  position  data  is  calculated  by  subtracting 
those  2  timestamps  and  recorded  into  a  log  file. 

d.  Stop  the  Position  Data  Sink  after  about  60  minutes. 

e.  Open  the  latency  log  file  and  display  the  results  graphically. 

Result: 

The  results  displayed  below  show  the  complete  position  VDM  creation,  broadcast 
and  consumption  time,  i.e.  end-to-end  latency  as  shown  in  Figure  7. 
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Figure  7:  VDM  end-to-end  time  for  Position  Service 

The  End-To-End  time  for  creating  position,  transmitted  and  sinking  the  VDM  ranges 
from  60,000  nsec  to  105,000  nsec  with  some  sporadic  peaks  up  to  140,000  nsec. 


Average  =  100,000nsec 
Standard  Deveiation  =  56.000nsec 


13.0  Conclusion: 

The  experiments  performed  in  the  VIDS  evaluated  the  VICTORY  Architecture 
Standards  1.0  interface  specifications  by  integrating  software  clients  and  services 
developed  to  the  specifications,  and  evaluated  the  resulting  functional  behavior  and 
performance.  In  conclusion,  the  Position  service  as  specified  for  the  1.0  standard  as 
implemented  in  the  VIDS  is 

o  Complete  and  unambiguous  of  the  individual  interface  specifications  of 
each  included  service. 
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o  The  resulting  functional  performance  of  this  reference  implementation 
with  the  representative  hardware  is  adequate  to  the  current  development 
needs  of  the  VICTORY  architecture. 

The  next  step  in  validating  the  service  is  determining  whether  interoperability  is 
achieved  when  multiple  implementers  develop  to  the  specifications.  These  tests  will 
be  conducted  in  the  VIDS  and  the  results  will  be  published. 


14.0  Open  Issues  for  Future  Testing: 

The  testing  completed  at  this  stage  included  only  the  verification  of  the  standard  for 
the  standalone  Position  Service.  Future  testing  will  include  interoperability  testing, 
system  level  testing  and  end-user  testing.  To  include  the  services  for  integrating  to 
end  user  applications  the  following  tests  will  be  done  and  the  ensuing  development 
will  be  recommended. 

o  Notification  to  the  user  and  the  application  when  the  GPS  signal  is  lost. 
Recommend  mitigation  actions  to  the  standard  specifications  or  at  the 
application  end. 

o  Test  and  report  the  limit  to  data  broadcast  frequency  i.e.  the  update 
period  and  limit  to  the  number  of  connections  (open  sockets)  to  the  data 
server  when  multiple  services  (Position,  DOT,  Orientation,  Threat,  RWS 
etc.)  are  implemented  on  the  same  server. 
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